Gradient-based 2D-to-3D Conversion for Soccer Videos

A wide spread adoption of 3D videos and technologies is hindered by the lack of high-quality 3D content. One promising solution to address this problem is to use automated 2D-to-3D conversion. However, current conversion methods, while general, produce low-quality results with artifacts that are not acceptable to many viewers. We address this problem by showing how to construct a high-quality, domain-specific conversion method for soccer videos. We propose a novel, data-driven method that generates stereoscopic frames by transferring depth information from similar frames in a database of 3D stereoscopic videos. Creating a database of 3D stereoscopic videos with accurate depth is, however, very difficult. One of the key findings in this paper is showing that computer generated content in current sports computer games can be used to generate high-quality 3D video reference database for 2D-to-3D conversion methods. Once we retrieve similar 3D video frames, our technique transfers depth gradients to the target frame while respecting object boundaries. It then computes depth maps from the gradients, and generates the output stereoscopic video. We implement our method and validate it by conducting user-studies that evaluate depth perception and visual comfort of the converted 3D videos. We show that our method produces high-quality 3D videos that are almost indistinguishable from videos shot by stereo cameras. In addition, our method significantly outperforms the current state-of-the-art method. For example, up to 20% improvement in the perceived depth is achieved by our method, which translates to improving the mean opinion score from Good to Excellent.Qatar Computing Research Institute-CSAIL PartnershipNational Science Foundation (U.S.) (Grant IIS-1111415

Anahita: A System for 3D Video Streaming with Depth Customization

Producing high-quality stereoscopic 3D content requires significantly more effort than preparing regular video footage. In order to assure good depth perception and visual comfort, 3D videos need to be carefully adjusted to specific viewing conditions before they are shown to viewers. While most stereoscopic 3D content is designed for viewing in movie theaters, where viewing conditions do not vary significantly, adapting the same content for viewing on home TV-sets, desktop displays, laptops, and mobile devices requires additional adjustments. To address this challenge, we propose a new system for 3D video streaming that provides automatic depth adjustments as one of its key features. Our system takes into account both the content and the display type in order to customize 3D videos and maximize their perceived quality. We propose a novel method for depth adjustment that is well-suited for videos of field sports such as soccer, football, and tennis. Our method is computationally efficient and it does not introduce any visual artifacts. We have implemented our 3D streaming system and conducted two user studies, which show: (i) adapting stereoscopic 3D videos for different displays is beneficial, and (ii) our proposed system can achieve up to 35% improvement in the perceived quality of the stereoscopic 3D content

Generating and streaming immersive sports video content

Stereoscopic 3D videos have already become popular in movie theaters with most productions being released in this format. More recently, with the availability of commodity Virtual Reality (VR) products, immersive video content is receiving even more interest. A wide spread adoption of immersive devices and displays is hindered by the lack of content that matches the user expectations. Producing immersive videos is far more costly and time-consuming than regular 2D videos, which makes it challenging and thus rarely attempted, especially for live events, such as sports games. In addition, immersive content needs to be adapted for viewing on different displays/devices. To address these challenges, we first propose a new system for 3D video streaming that provides automatic depth adjustments as one of its key features. Our system takes into account both the content and the display type in order to customize 3D videos and optimize the viewing experience. Our stereoscopic video streaming system was implemented, deployed and tested with real users. Results show that between 60% to 70% of the shots can benefit from our system and more than 25% depth enhancement can be achieved. Next, we propose a novel, data-driven method that converts 2D videos to 3D by transferring depth information from a database of similar 3D videos. Our method then reconstructs the depth map while ensuring temporal coherency using a spatio-temporal formulation of Poisson reconstruction. Results show that our method produces high-quality 3D videos that are almost indistinguishable from videos shot by stereo cameras, while achieving up to 20% improvement in the perceived depth compared to the current state-of-the-art method. Furthermore, we extend our work in the direction of VR, and propose using video feeds from regular broadcasting cameras to generate sports VR content. We generate a wide-angle panorama by utilizing the motion of the main camera. We then use various techniques to remove the parallax, align all video feeds, and overlay them on the panorama. Subjective studies show that our generated content provides an immersive experience similar to ground-truth content captured using a 360 camera, with most subjects rating their sense of presence from Good to Excellent